On the Value of User Preferences in Search-Based Software Engineering:
•
1 like•802 views
Tim Menzies, Abdel Salam Sayyad, Hany Ammar
Recommended
More Related Content
What's hot
Similar to On the Value of User Preferences in Search-Based Software Engineering:
Similar to On the Value of User Preferences in Search-Based Software Engineering: (20)
More from CS, NcState
More from CS, NcState (20)
Recently uploaded
Recently uploaded (20)
On the Value of User Preferences in Search-Based Software Engineering:
- 1. On#the#Value#of#User#Preferences#in# Search4Based#So7ware#Engineering:## A#Case#Study#in#So7ware#Product#Lines## # Tim#Menzies,## Abdel#Salam#Sayyad,# Hany#Ammar# # WVU,#USA# Nov’12## #
- 2. Sound#bites# • The#new#age#of#the#app# # • Stop#Nnkering#with#small#stuff# # • Enough#with#the#usual#suspects:## – NSGA4II,#SPEA2,#etc# # • If#preferences#maSer# – Then#the##best#opNmizer#understands#preferences#the#best# 2#
- 3. Roadmap# ① Feature(based,SE, ② Algorithms, ③ IBEA, ④ Tree,muta<on, ⑤ Conclusion, #
- 4. Roadmap# ① Feature(based,SE, ② Algorithms, ③ IBEA, ④ Tree,muta<on, ⑤ Conclusion, #
- 6. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,, e.g.#Microso7#office# • E.g.#Apple#app#store# 6#
- 7. The#Nmes,#they#are#a#changing# Olde,worlde:,, New,worlde:,, product(based,SE, app(based,SE,, • Vendors#tried#to#retain#their# • Smart#phones#and#tablet4 user#base#via#some# based#so7ware# complete#ecologies# # # • Users#choosing#many# numbers#of#small#apps#from# • One#so7ware#soluNon#for# different#vendors,## all#user#needs###(e.g.# – each#performing#a#specific# Microso7#Office).## small#task.# # # • Large,#complex,#so7ware# • Vendors#must#quickly#and# plaorms,## conNnually#reconfigure#apps## – To#retain#and#extend#their# – very#slow#to#change.## customer#base.# 7#
- 8. Feature–oriented#domain#analysis# • Feature#maps#=#a# lightweight#method#for# defining#a#space#of##opNons# • Product4line#configuraNons# • Defacto#standard#for# modeling#variability## 8#
- 9. Original#FODA#paper#:#2700+#citaNons# Half#since# 2007# 9#
- 10. hSp://www.splot4research.org/# 200+#models,##plus#an#instance#generator# 10#
- 12. Need#for#beSer#automaNon# • Such#complexity#needs#automated#support# – especially##feature#models##combined#with# • #user#preferences#and#prioriNes,#e.g.#cost#and#reliability.## Search#for#valid#products:## 9#state#of#the#art#theorem# provers##[Pohl,#ASE’11]# # Bad## scalability# And#these#were# “simple”#models# 12#
- 13. Diving##deeper# • Much#prior#work#explored## Nny#objecNve#spaces## – Two#or#three#objecNves# – Or,#higher#(but#only#for#small#models)# # # # • So7ware#engineering#=#navigaNng#compeNng#concerns# 1. That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤#100%)# The# usual# 2. That#offers#most#features# suspects# 3. Build#“stuff”#In#least#Nme# 4. That#we#have#used#most#before# 13# 5. Using#features#with#least#known#defects#
- 14. Roadmap# ① Feature(based,SE, ② Algorithms, ③ IBEA, ④ Tree,muta<on, ⑤ Conclusion, #
- 15. MOEA=#MulN4objecNve### evoluNonary#algorithms####################### • Repeat#Nll#happy#or#exhausted# – SelecNon#(cull#the#herd)# – Cross4over#(the#rude#bit)# – MutaNon#(stochasNc#jiggle)# 15#
- 16. Some#MOEA#ApplicaNon# Domain Application Types Control gas pipeline, pole balancing, missile evasion, pursuit Design semiconductor layout, aircraft design, keyboard configuration, communication networks Scheduling manufacturing, facility scheduling, resource allocation Robotics trajectory planning Machine Learning designing neural networks, improving classification algorithms, classifier systems Signal Processing filter design Game Playing poker, checkers, prisoner’s dilemma Combinatorial set covering, travelling salesman, routing, bin packing, Optimization graph colouring and partitioning
- 17. MOEA#for#Search4based#SE# TransformaNon #Cooper,#Ryan,#Schielke,#Subramanian,#FaNregun,#Williams# Requirements## #Bagnall,#Mansouri,#Zhang# Effort#predicNon# #Aguilar4Ruiz,#Burgess,#Dolado,#Lefley,#Shepperd## Management # #Alba,#Antoniol,#Chicano,#Di#Pentam#Greer,#Ruhe# Heap#allocaNon #Cohen,#Kooi,#Srisa4an## Regression#test #Li,#Yoo,#Elbaum,#Rothermel,#WalcoS,#Soffa,#Kampxamer## SOA## # # #Canfora,#Di#Penta,#Esposito,#Villani## Refactoring # #Antoniol,#Briand,#Cinneide,#O’Keeffe,#Merlo,#Seng,#TraS# Test#GeneraNon #Alba,#Binkley,#BoSaci,#Briand,#Chicano,#Clark,#Cohen,#Gutjahr,## ## # # #Harrold,#Holcombe,#Jones,#Korel,#Pargass,#Reformat,#Roper,#McMinn,# ## # # #Michael,#Sthamer,#Tracy,#Tonella,Xanthakis,#Xiao,#Wegener,#Wilkins# Maintenance # #Antoniol,#Lutz,#Di#Penta,#Madhavi,#Mancoridis,#Mitchell,#Swi7# Model#checking #Alba,#Chicano,#Godefroid# Probing # # #Cohen,#Elbaum## UIOs# # # #Derderian,#Guo,#Hierons# So#study#FODA4# Comprehension #Gold,#Li,#Mahdavi# to#learn#how#to# Protocols# # #Alba,#Clark,#Jacob,#Troya# improve#these# Component#sel #Baker,#SkalioNs,#Steinhofel,#Yoo# tasks.# Agent#Oriented #Haas,#Peysakhov,#Sinclair,#Shami,#Mancoridis# 17#
- 19. The#Pareto#FronNer# • Mutants#=#<D,O>#=#<decisions,#objecNves># – E.g.#car# • Decisions:#color#of#car,#number#of#cylinders,#number#of#wheels# • ObjecNves:#miles#per#hour,#cost##(objecNves#may#complete)# – E.g.#learning#formula# • Decisions:#what#variables#and#constants#to#use# • ObjecNves:#model#simplicity#vs#effecNveness#(objecNves#may# complete)# • Pareto#fronNer:#select#the#non4dominated#mutants# – X#dominates#Y## • if#for#all#objecNves,##X#is#never#any#worse##than#Y# • If#for#one#objecNve,#X#beSer#than#Y# 19#
- 20. Once#you#know#fronNer# •#Select#from#here# Issues:# •#Ignore#here# 1) Spread# 2) Hypervolume# 3) ComputaNonal#cost:##“g”##generaNons,#M#mutants,#O(gM2)# 20#
- 21. The#usual#suspects:# ##=#NSGA4II### #=#SPEA2# In#this#case,#NSGA4II# gets#more#spread# Combines#N#objecNves## to#one##with#some## weighNng#scheme# 21#
- 22. Some#details#on#the#usual#suspects# NSGA(II,, SPEA2,, • Is#a#geneNc#algorithm# • Is#a#geneNc#algorithm# # # • Changes#the#definiNon#of# • Non4dominated#sort# “dominaNon”# – HeurisNc#way#to#fast#group# – SPEA#(version#1)#scored#mutants#by# mutants#into#bands# how#many#others#they#dominated# # – Got#confused#by#overlaps#in#the# dominaNon#sets# • Crowd#pruning##via#approximate# # hypercube#around#each#mutant:# • SPEA#(version#2):# O(Onlogn)# – Adds#a#“local#density#factor”#to#the# dominaNon#weight# 3# – Mutants#in#dense#areas#valued## 2# more# ## 1# • SPEA2#beSer#than#SPEA1# 2# 22#
- 23. Any#number#of#opNmizaNons## to#tradiNonal#GAs# • The#history#of#MOEAs#in#the#last# • DifferenNal#evoluNon#(Storn#1996)# 15#years#is# – Mutate#by#interpolaNons#between# – OpNmize#via#hybrid#GA#+#other# exisNng#mutants# search#method# # – For#x#in#mutants## ####y#=#any1#+#extrapolate(any3#–#any2)# • Local#search:## ####if#y#dominates#x#then#x#=#y# – before#select,#do#a#liSle#simulated# # annealing#on#X%#of#the#populaNon# # • Cellular#automata,# • ScaSer#search#(Glover’s#next# • #Ant#colony#opNmizaNon,#### generaNon#tabu#search)# – Includes#a#liSle#local#search# • Bayesian#staNsNcs#to#bias#the# # mutaNon,## • ParNcle#swam#opNmizaNon# • Etc## – May#do#as#well#as#scaSer#search##(Yin# • etc# and#Glover#2007)# 23#
- 24. Roadmap# ① Feature(based,SE, ② Algorithms, ③ IBEA, ④ Tree,muta<on, ⑤ Conclusion, #
- 25. Three#groups#of#Algorithms# DominaNon# DominaNon# Is#a#binary# PSO# Is#a#conNnuous# concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 25#
- 26. IBEA# • Bo#smarts#anywhere#except#in#the#exploraNon#of#preferences# • I(x1,x2):# – Least#adjust#objecNve#scores#such#that#x1#dominates#x2# • Repeat#Nll#just#a#few#le7# – Score#each#instance#x1##buy#summing#its#“I”#to#everyone#else# # K=# # 0.05# # # # # – Sort#all#instances#by#F# – Delete#worst# • Then,#standard#GA#(cross4over,#mutaNon)#on#the#survivors# ## 26#
- 27. Case#studies# Data#from#hSp://www.splot4research.org/# Algorithms#from#jMetal:#hSp://jmetal.sourceforge.net/## Cross4tree# constraints# 27#
- 28. 4#studies:# Bi,#tri,#quad,#five4#objecNves# So7ware#engineering#=#navigaNng#compeNng#concerns# 1. That#saNsfies#most#domain#constraints#(0#≤###violaNons#≤# 100%)# 2. That#offers#most#features# 3. Build#“stuff”#In#least#Nme# 4. That#we#have#used#most#before# 5. Using#features#with#least## known#defects# # # Binary#objecNves#=#1,2# Tri4objecNve#########=#1,2,3# Quad4objecNve####=#1,2,3,4# Five4objecNve######=#1,2,3,4,5# 28#
- 36. Why#is#this#interesNng?# Other,MOEAs, IBEA, • The#usual#suspects#are#widely,# • Rather#stupid#on#those# uncriNcally#used#in#many#MOEA# applicaNons# internal#tricks# – E.g.#especially#NSGA4II#and#SPEA2# – Just#does#a#ye#olde#crossover# • Focused#on#internal#algorithmic# mutate#GA# tricks# – Plus:#aggressive#exploraNon# – Techniques#for# of#the#preference#space# • #improving#spread## • Improving#HV# • And#the#net#effect#of#all# • Avoid#overlaps#in#cross4over#of# dominated#space# those#differences# • etc# – BeSer#spreads# • And#the#net#effect#of#all#those# – BeSer#HV# differences?# – Not#much# – Fewer#constraint#violaNons# Conclusion:# preference#is#power# 36#
- 37. Roadmap# ① Feature(based,SE, ② Algorithms, ③ IBEA, ④ Tree,muta<on, ⑤ Conclusion, #
- 38. What#about#non4MOEA#soluNons?# DominaNon# DominaNon# Is#a#binary# PSO# Is#a#conNnuous# concept# concept# DE# ScaSer## IBEA# Spea2# search# Aggressive# exploraNon# Indicator4based## of#preference## SA# evoluNonary# space# Nsga4II# algorithms# mocell# Z3# SMT#solvers# 38#
- 39. Ethan’s#complaint# • So7ware#engineer## designs#are#o7en## nested#hierarchical## constraints# • Ethan#Jackson,#Microso7,## advocate#for#the#Z3#SMT#solver:# – Why#mutate#at#random,#then#check#for# constraint?# – BeSer#to#drive#the#mutaNons#by#the# constraints?# 39#
- 40. Dump#MOEAs?## Move#to#more#logical#forms?# Pro:, Con:, move,to,,say,,SMT,solvers, stay,with,MOEA, • Next#generaNon#of# • ImplementaNon#complexity# algorithm# less# – The#next#big#thing# • More#tools# • BeSer#to#drive#the# • Easier#modificaNon#and# mutaNons#by#the# experimentaNon# constraints?# • Models#can#be#expressed# anyway#you#like# 40#
- 41. Tree#mutaNon# • Work#in#progress# • Simple#adaptaNon#of#current#MOEAs#for#systems# of#hierarchical#constraints# # if#rand(0,1)<#mutaNon_probability:### #########Don't#mutate#if#you're#violaNng#one#of#the#rules:# # #1)#if#deselecNng#root#feature# # #2)#if#selecNng#feature#whose## ##########################parent#is#not#selected# # #3)#if#deselecNng#feature#that# #########################another#selected#feature#requires# # #4)#if#group#cardinality#violaNon# #else:# # #flip#this#bit# # # ##### # # #if#selecNng#(turning#on)#a#feature#then# # # # #turn#on#children### IBEA# # # #else#if#deselecNng#(turning#off)# stabilizes#70# #######################################feature#then:# Nmes#faster# # # ######### #########turn#off#all#children# 41#
- 42. Tree#mutaNon#preserves## domain#constraints# So#what## case#for# SMT?# 42#
- 43. Roadmap# ① Feature(based,SE, ② Algorithms, ③ IBEA, ④ Tree,muta<on, ⑤ Conclusion, #
- 44. Sound#bites# • The#new#age#of#the#app# – In,this,new,worlde:,,use,FODA, (feature(oriented,domain,analysis), • Stop#Nnkering#with#small#stuff## – Many,MOEAs,have,strikingly,, similar,performance, • Enough#with#the#usual#suspects:## – NSGA4II,#SPEA2,#etc# – Too,much,uncri<cal,applica<on,of,these,algorithms, • If#preferences#maSer# – Then#the##best#opNmizer#understands##preferences#the#best# – IBEA:,aggressive,preference,explora<on, – Tree,muta<on:,respect,your,domain, 44#